Interaction Context-aware Modalities and Multimodal Fusion for Accessing Web Services

These days, a human-controlled multimodal system equipped with multimodal interfaces is possible, allowing for a more natural and more efficient interaction between man and machine. In such a system, users can take advantage of the modalities to communicate or exchange information with applications. The use of multimodal applications, integrated with natural modalities, is an effective solution for users who would like to access ubiquitous applications such as web services. The novelty of this work is that all modalities that are made available to the user to access web services are already found to be suitable to the user’s current situation. By suitablity, we mean these are optimal modalities – found to be suitable to the user’s interaction context (i.e. the combined context of the user, his environment and his computing system) and media devices are available to support them. These modalities can be invoked for the data input/output by the user to access web service using a semantic combination of modalities, called “multimodal fusion”. While current state-of-the-art uses two (on rare cases, three) predefined modalities, our approach allows an unlimited number of concurrent modalities. This approach gives user some flexibility to use the modalities that he sees fit for his situation and comfortable with it. The description of the detection of optimal modality as well as the fusion process, together with sample application are presented in this paper.